As society becomes increasingly dependent on computers, it becomes increasingly important to detect and eliminate errors in the software executed by the computers. To detect errors, software developers typically run the software through a series of tests.
Software tests may be performed manually or automatically. Manual testing requires a user to enter test input during the testing process. As the user enters test input, the user notes how the computer system responds to the test input. If the computer system responds as expected, then the software passes the test. Otherwise, the software does not pass the test. Manual testing has the disadvantage that it involves a great deal of time and human resources. Further, repeat tests require approximately the same amount of time to perform as the original tests. Finally, notes made by the user during testing are often the only documentation of which tests were performed and what the results were.
For automated testing, a user prepares an input file with test input. The software to be tested reads test input from the input file and generates output based on the test input. The output can then be compared to expected output. If the actual output matches the expected output, then the software passes the test, otherwise it fails. Automated testing has numerous benefits over manual testing. For example, once a test input file and an expected output file have been prepared, virtually no additional user time is required to perform repeat tests. Further, the input test file serves to document which tests are run, and the generated output file serves to document the results of the tests.
The results of automated tests may be affected by the system environment in which the testing occurs. For example, two software programs compiled from the same source code may generate different outputs based on the same input if different compiler options were used to compile the two software programs. Testing may also be affected by the value of other environment parameters, such as the value of the "PATH" variable that is in effect when a testing operation is performed.
Consequently, testing operations that do not take into account the computer environment at the time of testing may yield inaccurate or inconsistent results. Therefore, it is clearly desirable to provide a method and apparatus for testing software that allows the tester to select different system environments. It is further desirable to provide a method and apparatus that establishes predefined system environments as part of an automated testing process.